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Summary 

Perspective synthetic displays that supplement, or 
supplant, the optical windows traditionally used for 
guidance and control of aircraft are accompanied 
by potentially significant human factors problems 
related to the optical geometric conformality of 
the display. Such geometric conformality is 
broken when optical features are not in the 
location they would be if directly viewed through 
a window. This often occurs when the scene is 
relayed or generated from a location different 
from the pilot’s eyepoint. However, assuming no 
large visual/vestibular effects, a pilot can often 
learn to use such a display very effectively. 
Important problems may arise, however, when 
display accuracy or consistency is compromised, 
and this can usually be related to geometrical 
discrepancies between how the synthetic visual 
scene behaves and how the visual scene through a 
window behaves. In addition to these issues, this 
paper examines the potentially critical problem of 
the disorientation that can arise when both a 
synthetic display and a real window are present in 
a flight deck, and no consistent visual 
interpretation is available. 

Introduction 

Synthetic perspective displays stand in contrast to 
most aircraft flight deck displays which provide 
specific vehicle-state variables and guidance cues, 
and which supply the pilot with general 
situational/spatial awareness. Examples of such 
instruments are the traditional attitude direction 
indicator, turn coordinator, airspeed indicator, 
heading indicator, and vertical speed indicator, 
with these being displayed either on panel- 
mounted displays (PMDs), or within a head-up 
display (HUD). These instruments have been 
developed primarily by controls engineers who 
typically tailor these control-display concepts in a 
series of simulations using measured performance 
and subjective handling qualities ratings (ref. 1). 
Although the aviation community is very 
comfortable with this type of instrument suite 
(some of which are based on quite advanced 
concepts (refs. 2 and 3)), a growing body of 
literature generated by human factors researchers 


suggests that the naturally integrated and non- 
symbolic nature of the perspective scene as 
sampled through a flight deck window is often, but 
not always, a superior source of information, 
providing the pilot with information in a very 
familiar format for guidance and control, and 
giving good spatial and temporal preview of the 
intended path.(refs. 4 and 5). 

Further, conventional avionic displays such as 
described above are of really very limited utility 
during aircraft surface operations, where the 
perspective scene sampled through a flight deck 
window is the primary source of information for 
ground control and guidance. That is, other than 
paper maps and perhaps a ground speed display, 
there are almost no instruments or displays on the 
flight deck useful for ground guidance, control, 
and object detection and avoidance. Nor has 
anyone suggested a display that permits a pilot to 
directly guide and control surface movement as 
well as he can do this with flight deck windows on 
a clear day. 

However, as a result of the acknowledged value of 
naturalistic perspective scenes for guidance and 
control, and as a result of the growing body of 
literature on the advantage of integrated displays, 
there has been an effort to create flight deck 
displays that have an integrated perspective format 
(refs. 6 and 7). These displays, which attempt to 
partially replace or supplement the perspective 
scene available through the window of the aircraft, 
can he termed virtual windows since, like 
windows, they approximate the boundaries or 
edges of an aperture, and thus approximate a 
window into a virtual space (i.e. a simulated 
projection). Additional guidance for this effort has 
also come from the work on visual cues used for 
vehicle control (refs. 8, 9, 10, 11, 12, 13, 14), and 
simulation cueing requirements (refs. 15 and 16). 

Unfortunately, while there has been a great deal of 
research on the desirable properties of such 
integrated 3-D displays, there has been little 
examination of the human factors problems that 
may arise when one attempts to incorporate such 
displays into a real flight deck. This paper is an 
examination of some selected issues in this area 
that the authors have encountered in the course of 



their work on flight deck displays. The 
organization of this paper will be to first briefly 
describe some of the salient properties of real 
flight deck windows, and then to very briefly 
describe some of the technologies that are 
available at present to support the implementation 
of virtual windows. Next there will be a discussion 
of the human factors concerns that stem from 
differences between displays generated using these 
technologies and the visual display available with a 
real window. 


Properties of Windows 

To the extent that a rendered perspective scene is 
an approximation of a true window, we may ask: 
“what are the characteristics of the true window 
that may or may not be present in a virtual 
window?” The following is a short list of relevant 
window properties. Many other, perhaps equally 
important, properties could be described. However, 
these factors are especially relevant for the issues 
raised in the remainder of the paper. 

First, there is the correspondence between the 
visual content of a true window and the visual 
content of a rendered space. Ignoring 
considerations of glare and a dirty window, a true 
window faithfully shows what is on the other side 
of it, including other aircraft, atmospheric 
conditions (weather), and the general airport 
layout. Furthermore, in low-visibility conditions it 
does not show much of the world. This is in 
contrast to a virtual window which, depending 
upon the technology behind it, may show only 
part of what is in the world behind it, and may also 
introduce elements that are not in the world. For 
example, a perspective display that is solely 
derived from a computer-generated imagery 
(CGI) database will not be subject to the visibility 
limitations of weather, and may depict useful 
signage not actually present in the real world, but 
will also not capture elements in the world that are 
not a part of this database (e.g. an approaching 
vehicle). 

Second, the perspective scene sampled through a 
window always obeys optical laws relating the 
location of the objects in the world, to the location 


of the window on the aircraft, and to the location 
of the observers on the flight deck. For example, 
the field of view (FOV) and field of regard (FOR), 
made available by the window, and the optical 
location of a particular object within the window, 
are always specific to the location of the 
observer’s eye(s) (i.e. are different for every 
potential observer location on the flight deck). For 
the case of multiple flight deck windows this 
means that there is a unique set of correct views 
through these windows for every location on the 
flight deck. This is in contrast to synthetic 
perspective displays which may not all be drawn 
from the single perspective point of view of the 
pilot. 

Third, the optical transformations or changes 
within true windows are always specifically linked 
to the vestibular stimulation of the observer. Thus 
movements of the scene within, the windows always 
specify the movement of the pilot’s head within 
the flight deck, or of the flight deck itself. This is 
in contrast to synthetic perspective displays that 
may not change when the pilot moves his head, or 
which may pan and zoom without any movement 
of the aircraft or the pilot. 

Fourth, moving the eyes relative to a window 
produces several salient optical changes. One 
change is a redefinition of the viewing volume (the 
portions of the outside world that can be seen). 
When you move closer to the window more of the 
outside world is visible; when you move left, right, 
up or down new areas of the outside world are 
exposed to. view and some of the old areas are 
occluded by the window boundary (see Figure 1). 
This reflects changes in the size of the field of 
view (FOV), and in the areas of the field of regard 
(FOR) which fill the FOV. For example, Figure 1 
also shows that when you move closer, the FOV 
spanned by the window expands and is filled in by 
a larger portion of the FOR. On the other hand, 
when you move up, down, left, or right with 
respect to the window the viewed portion of the 
FOR may change substantially while the FOV may 
not change by a great amount. 

Fifth, windows provide the appropriate stimulation 
for binocular vision, and thus for stereo vision and 
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binocular vergence. Furthermore, since an 
observer’s (or pilot’s) eyes are laterally separated, 
the viewing volumes for the two eyes do not 
exactly overlap (binocular disparity), and the 
effective FOR is increased. Finally, since windows 
provide a true view in depth, they provide the 
appropriate stimulus for the ocular 
accommodation response. 

Finally, it is very important to note that there is no 
temporal lag induced by transmission of light 
through a window. All changes or transformations 
of optical patterns occur instantaneously, or in 
“real time”. This is in contrast to synthetic 
displays which always possess some degree of lag. 

Display Technologies 

We now need to briefly consider the nature of 
specific display technologies, since these will 
define both the capabilities and the limitations with 
regard to replicating the perspective scenes 
sampled through a true window. In doing this, we 
will ignore specific implementations and focus as 
generically as possible. Five such generic 


technologies have been identified below. 
Information Format 

First, there is the question of the format of the 
information that can be used. The two main types 
are viewpoint-based and coordinate-based. 
Viewpoint-based information is fundamentally 2- 
D, and comes primarily from cameras (electro- 
optical, infra-red, etc.) in the form of an image. 
This type of format does not directly contain 
information about the 3-D spatial structure or 
location of the any of the imaged objects, but 
instead represents image characteristics (hue, 
saturation, brightness) in an image-coordinate 
space. Viewpoint-based systems typically can only 
generate imagery that is appropriate for a 
viewpoint (eye location) that is coincident with the 
location of the sensor (e.g. camera or FLIR), 
although image-warping techniques have been 
proposed that, within limits, can modify the image 
so it approximates the view from another (usually 
nearby) viewpoint (ref. 17). 

Coordinate-based information, on the other hand, 
is fundamentally 3-D, and may be obtained from a 
database describing the layout of objects and 
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Figure 1. View out of three contiguous 2 ft. x 2 ft. windows. The scene contains a taxiway intersection 
367 ft. from the observer, and a 200 ft. aircraft on the crossing taxiway also 367 ft. from the 
intersection. The depicted incockpit viewing locations are 1 ft. and 2 ft. from the windows; and the 
Captain’s location (1 ft. left of center), the First Officer’s location (1 ft right of center), and a center 
point equidistant between the Captain and the First Officer. 
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surfaces (such as GGI database) or it may be 
obtained from sensor systems that can determine 
the range, elevation, and azimuth of surfaces that 
lie within the line of sight (such as RADAR, or 
some types of artificial vision systems that process 
the optical 2-D imagery). When coordinate-based 
systems are used, they can generate imagery that is 
appropriate for a variety of eye locations, since 
they store the information in a viewpoint 
independent manner. However, when the 
coordinate information comes from a sensor, and 
not a CGI database, there is no coordinate 
information for objects and surfaces that are not 
within the line of sight. Thus the ability to 
transform viewpoints is still limited. 

Finally, mixed format systems have also been 
proposed. These systems attempt to capitalize on 
the relative strengths of the two technologies, 
fusing the information provided by optical and 
infrared cameras, CGI databases, and RADAR to 
produce an enhanced image. However, such 
systems still are very much in the research and 
development stage. 

HMD vs. PMD 

Second, there are two main types of physical 
displays: helmet-mounted displays (HMDs) and 
panel-mounted displays (PMDs). A PMD is any 
display that is located in a fixed location within the 
flight deck. An HMD is any display that is worn at 
a fixed location relative to the eye, and may be 
totally or partially immersive. The display of a 
totally immersive HMD will cover the observer’s 
entire effective FOV. Partially immersive HMDs 
have been designed to allow pilots to see the other 
displays and controls within the flight deck. These 
partially immersive displays are either see-through 
(see following description of display transparency) 
or only cover part of the observer’s effective FOV 
(e.g. may only cover one eye, or only cover the 
upper part of the FOV, leaving the observer free to 
look underneath the display and see the control 
panel). 

Headtracking 

Third, displays can be head-tracked or not head- 
tracked. An HMD usually requires some form of 
head-tracking so that the display is always 


optically correct for the head location and 
orientation. Some HMDs are not of this type, 
displaying text or other symbolic information, but 
these are not of interest in the present context. 
While PMDs have typically not been coupled with 
a head-tracker, this can be done in order to 
produce optical transformations that mimic those 
occurring in a real window (ref. 18). 

Transparency 

Fourth, displays can be either opaque or “see- 
through”. Opaque PMDs and HMDs are the more 
common displays. An example of a see-through 
PMD is a HUD. Such see-through HMDs 
(displayed on a glass monocle) have been used to 
present Apache helicopter pilots with both 
symbolic and sensor-derived perspective HUD 
displays useful for guidance, control, and for 
aiming weapons. Surface taxi applications are 
being presently explored that Would use see- 
through HUDs with scene-linked perspective 
symbology (ref. 19). 

Binocular vs. Biocular 

Fifth, displays can be either binocular (presenting 
different images to the two eyes) or biocular. Both 
binocular and biocular imagery can be presented 
in two ways. Binocular imagery can either be 
sensitive or insensitive to head orientation. When 
the display is insensitive to head orientation, the 
standard method of generating stereo displays 
presents images to each eye that reflect the 
expected lateral displacement of the eyes along the 
purely horizontal axis. That is, this display 
assumes that the head is being held in a level 
position (i.e. not tilted to the left or right). The 
advantage of this type of display is that it does not 
require head-tracking. However, if the observer 
tilts his or her head from side to side, then this 
changes the orientation of the axis along which 
disparity is defined. In addition, the display will be 
incorrect if the observer moves his or her head out 
of the correct location, or “eyebox”, designated 
for the display. Therefore, only a system that 
calculates or obtains images that reflect head 
location and head roll will accurately capture these 
aspects of stereo vision. 
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Biocular displays also come in two types. There 
are displays that show an image upon a picture 
plane, with the two eyes sampling this same image 
from slightly different perspectives. This is the . 
normal case of picture viewing, and is one of the 
things that allows us to know that we are viewing a 
picture and not a real scene. Next there are 
displays that show the same image to both eyes, 
thus specifying that the image is somewhere 
behind the true picture plane. This can be 
accomplished with two methods. One method uses 
a head-mounted display (HMD) and presents the 
same image to the two eyes, with the image at 
optical infinity when no convergence is present. 
The other method uses a panel-mounted 
collimated display, with the image at optical 
infinity when the display is fully collimated. 

Human Factors Criteria 

The description of true windows can then guide 
our examination of displays as virtual windows. 
While limitations in available technology may 
make it impossible to replicate fully the properties 
of a true window, they do not preclude the 
generation of a virtual window that is useful and 
safely preserves many important perceptual 
properties of a window. But a successful 
implementation will require a methodical 
evaluation of how well the virtual window 
preserves these properties. This evaluation, in turn, 
should characterize virtual window adequacy with 
respect to a display’s functional utility, and any 
physiological side effects produced by the use of 
the display. A display’s functional utility reflects 
how well the pilot can use the optical information 
provided by the window to control and guide the 
aircraft, and to perform the required monitoring 
functions (e.g., target detection and weather 
monitoring). This is impacted both by the quality 
of the information contained in the display, and its 
compellingness or salience. Information quality 
reflects how accurate and complete the needed 
information is. Compellingness, or salience, 
reflects how likely a person is to pick up and use 
the information. A highly compelling display 
attribute can be a two-edged sword, however, 
depending upon the accuracy of the display 
attribute. 


Simulation and telerobotic applications have also 
demonstrated the importance of the physiological 
side effects produced when synthetic perspective 
displays are used. These side effects all generally 
fall under the category of simulator sickness (e.g. 
nausea, headaches, and disorientation) which can 
accompany displays where there is cue conflict. 
Simulator sickness is most often seen in conflicts 
between visual and vestibular information (e.g. a 
simulator presents a visual display which specifies 
self-motion, but accompanies this with either no 
platform motion, or incorrect platform motion). 

Finally, in order to guide this evaluation we need 
some principled method by which to organize our 
comparison of the physical fidelity of the virtual 
window with respect to a real window. Three 
attributes may be used to form the primary bases 
for evaluating this fidelity: geometric 
conformality, dynamic conformality, and image 
quality. This paper, however, is mainly concerned 
with issues of display conformality and will 
therefore examine only issues relevant to the first 
two issues. 

Geometric Conformality 

Perfect geometric conformality is achieved when 
the locations of the imaged objects within the 
virtual space register exactly with the optical 
locations of the real objects as directly viewed by 
the observer. There are dangers to violating 
geometric conformality, but there may often be 
compelling reasons to do so. For example, there 
may be a requirement for a larger effective FOV 
while keeping the cost and weight of the display to 
a minimum hence necessitating minification. In 
addition, the desire for other instrumentation may 
also lead to limitations on the amount and location 
of flight deck territory given to the perspective 
display. This directly clashes with the needs of 
pilots who, during surface operations, are 
monitoring a very wide FOV for approaching 
ground traffic. Geometric conformality violations 
such as a minified display, which compresses the 
FOV (and thereby change the appropriate eyebox 
location), or a slewable display which extends the 
effective FOR to the left and right (and thereby 
simulates head movements without corresponding 
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vestibular stimulation), appear very attractive 
options under such circumstances. 

There may also be a desire to use cameras to 
supply the imagery since this eliminates, or 
mitigates, the need for: 1) an on-board CGI system 
for high-fidelity graphical rendering; 2) sensor 
technologies which automatically detect, identify, 
and/or localize transient objects or situations, (e.g. 
aircraft, ground vehicles, and weather conditions); 
and 3) complex and frequently updated airport 
CGI databases. These frequent updates are needed 
since the available runways and taxiways are 
constantly changing at airports due to construction 
and maintenance. (For example, on a site visit to 
Atlanta’s Hartsfield International Airport the first 
author was told that significant changes can occur 
on a daily basis. Pilots are alerted to these changes 
both by published Airmen Advisories and signage 
on the airport surface.) However, as mentioned 
earlier, the camera supplies imagery that is 
appropriate for an eyepoint at the location of the 
camera, not the eyepoint of the pilot, and thus 
significant distortions in geometric conformality 
may ensue; 

Most of the ways in which violations of geometric 
conformality in perspective displays affect 
perception and performance have not been 
studied. This is especially true for aerospace 
applications of perspective displays, since the main 
impetus in the most studied application area, flight 
simulation, has been to provide this conformality 
and not study its absence. However, it may not be 
possible to avoid significant compromises in 
geometric conformality when placing perspective 
displays within flight decks. Initially, critical 
geometric conformality disruptions can be 
classified with respect to (at least) the following 
seven issues: 

1) Displacement of the nominal display 

viewpoint from the nominal eyepoint from 

which the display is to be viewed 

2) Displacement of the pilot’s head and eyes 

from the nominal eyepoint 


3) Inappropriate binocular disparity due to 
lack of stereo imagery or to use of 
hyperstereo 

4) Displacement of display surface from its 
appropriate location 

5) Distortions of depth information due to 
inappropriate convergence and 
accommodation cues 

6) Display minification 

7) Degree and/or type of dissimilarity 
between display and the view through optical 
windows 

Viewpoint Displacement 

This issue arises when the scene is drawn or 
generated from a location different from the 
pilot’s eyepoint. As mentioned earlier, this is an 
inevitable result when a sensor (e.g. a camera) is 
used to generate viewpoint specific imagery. In 
that case the distance of the camera (the nominal 
display viewpoint) from the pilot’s head (the 
nominal eyepoint) introduces a conformality 
breakage. In the absence of other windows on the 
flight deck, the major concern here is the possible 
visual-vestibular conflicts that will arise if the 
sensor undergoes motions that are difference in 
magnitude or type from the motion the pilot 
senses vestibularly. One place where this is a 
serious concern is during any maneuver where the 



Figure 2. Lateral velocity components of pilot and 
tail sensor motion during turns. 
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aircraft undergoes a change in orientation, and 
where the pilot and the sensor are at different 
distances from the center of rotation. In such a 
case the translational velocities and accelerations 
may be very different between the two. An 
extreme example, shown in Figure 2, is where the 
camera is significantly to the rear of the pivot 
point of the vehicle, while the pilot is significantly 
in front of the pivot point. Under test 
circumstances the pilot and the camera undergo 
opposite signed lateral accelerations. Another 
example occurs when there is differential sensor 
and pilot movement due to elastic aircraft flexing. 

Pilot Eyepoint Displacement 

This issue arises when the pilot’s head position is 
not coincident with the display’s design eyebox. 
As noted earlier, when a pilot changes his or her 
head position within the flight deck in order to get 
clear views of potentially approaching traffic, both 
inter-ocular parallax at window edges, as well as a 
change in what is seen, occurs. Even when there is 
no significant head motion, the pilot’s head and 
copilot’s head are positioned so as to sample 
substantially different areas of the FOR. For a 
display that is not sensitive to head position, and/or 
in situations where the two pilots do not have 
displays sensitive to their own head position, it 
may not be possible to get the appropriate view. 
Figure 1 shows how the ability to see traffic 
approaching an intersection can be affected when 
both pilots are forced to view a display that is 
conformal for a single eyebox centered on one of 
the pilots, or centered between the two pilots. This 
figure shows that a display centered on the 
nominal Captain’s position would leave the 
approaching ground traffic out of sight to the left. 
However, for virtual windows that are headslaved, 
the Captain could bring the displayed traffic into 
his view by moving his head forward, or to the 
right. In addition to the decreased FOR for the 
crew, not slaving the eyebox to eyepoint will also 
cause distortions in the flow patterns within the 
display. For example, if a crew member’s head is 
left of the correct eyebox location during straight- 
ahead motion, the center of optical expansion will 
no longer be located (correctly) directly in front, 
but instead it will lie off to a side (see Figure 1). 
Other optical anomalies are bound to be present 


also, including probable significant impact to the 
compellingness of the display. However, the 
impact on vehicle control and traffic/object 
avoidance remains to be assessed. 

For eyepoint conformality, two different displays 
are also needed to accommodate the range of 
different eyepoints required by a pilot and copilot, 
either of whom may be moving his or her head. 
Two displays would certainly be needed if an 
adverse impact on control was found for cross 
cockpit viewing of a display correct for the other 
pilot’s eyebox (or an eyepoint between the two 
pilots). In addition, two displays would be 
absolutely necessary if the eyebox was head- 
slaved, since viewing a display that transformed 
due to the head movements of another pilot would 
be extremely disorienting. The critical problem in 
implementing dual displays is making them 
viewable only by the appropriate crew member. 
Several options could be available. First, the cabin 
could be designed such that the pilot and copilots 
displays are physically screened from the other 
member’s view by some type of barrier* However, 
due to the limited space in the cockpit, this option 
places significant limits on the size of the display 
each of them could see (and therefore on display 
FOV). Another would be to provide each crew 
member with an HMD. This solves the viewability 
problem, but it is not clear if the needed display 
resolutions are available in HMDs (or if an HMD 
would interfere with necessary information 
exchanges between the crew members). Finally, 
there are technologies which may allow two 
eyepoint-specific displays to be presented on the 
same display surface. These approaches take 
advantage of the technologies used to provide 
stereo vision, and give each of the two pilots one 
of the two eyepoints. These technologies include: 
LCD shutter glasses (i.e., the view for each pilot is 
temporally interlaced); lenticular lensed displays 
(i.e:, the view for each pilot is spatially interlaced); 
and glasses with polarized filters (with the light for 
the two pilots’ views filtered orthogonally). While 
these approaches also solve the viewability 
problem, they impose the burdens of decreased 
resolution, and increased image generation 
load/faster update rates. 
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Finally, information format is another 
consideration when trying to incorporate head- 
movements in a virtual window. While 
transformations due to head movement can (in 
principle) be captured easily by systems using 
coordinate-based information formats (e.g. 
computer-generated imagery), systems based upon 
viewpoint-based formats would need to translate a 
camera in 3 axes to fully mimic head movements. 
On the other hand, it is possible that some of the 
viewpoint warping techniques (or camera zoom 
and pan) may be sufficient for a compelling and 
effective display. 

Stereo Vision Anomalies 
This issue arises whenever a display does not 
provide for stereo viewing, or provides 
hyperstereo. The lack of stereo vision generates 
two potentially problematic effects. First, a 
binocular view through a window yields significant 
inter-ocular parallax at the window edges and an 
increased FOR; a non-stereo display provides 
neither. The lack of the increased FOR impacts the 
functional utility of the display in the same 
manner as described in the issue above, while the 
lack of parallax at the window edges has its impact 
oh the compellingness of the display. The use of 
hyperstereo (using two viewpoints or camera 
locations separated by a much greater distance 
than that between the two eyes) has its own 
benefits and problems. On the positive side, the 
appreciation of relative depth is greatly enhanced, 
as is the compellingness of the display. On the 
downside is the fact that people are accustomed to 
their natural eye separation, and thus hyperstereo 
effectively miniaturizes the world (i.e. the 
increased stereo disparity is consistent with viewing 
a much smaller world). This can lead to 
misjudgments of size and depth, and could 
potentially lead to vestibular-visual side-effects 
when the visual changes appear to specify a small 
motion, while the vestibular system detects a much 
larger motion. 

Second, for objects that are within range of stereo 
vision, a stereo image provides depth or distance 
information. The importance of this property is a 
direct function of the distance between the window 
and the objects. The potentially most important 


situation where stereo vision could be valuable is 
during the approach to a gate. Here stereo vision 
could provide important distance information if 
the virtual window occupies a (virtual) position 
close to the nose of the craft. Again, the lack of 
stereo could also adversely impact the 
compellingness of the display. 

Display Displacement 

This issue arises when a display is not correctly 
located on the flight deck. A forward-looking 
display should be in front of a pilot; the horizon 
line should be at eye level. Otherwise the display is 
misplaced. During surface operations the 
misplacement of the horizon line, in combination 
with vestibular cues given by the vector of gravity, 
generate a cue conflict that may lead to 
physiological side effects. This possibility should 
be examined particularly for cases where there are 
significant longitudinal accelerations/decelerations, 
because these are known to create pitch attitude 
illusions. In such instances a pilot may have 
trouble determining when the illusion of pitch 
change is due to a misplaced horizon, and when it 
is due to acceleration. 

Inappropriate Convergence/Accommodation Cues 

This issue refers to the fact that the degree of 
visual convergence and the amount of 
accommodation, when focusing on an object 
through a window, is informative about the 
distance of that object. When focusing on a 
display surface, these cues specify that the object is 
very close and two dimensional. Often, an attempt 
will be made to eliminate the perception of the 
display surface by using collimating optics which 
generate accommodation and convergence cues 
that indicate that everything is at some fixed 
distance (the exact distance depends upon the 
design of the collimating lens). While coilimation 
may be added to generate the perception of a deep 
scene, compelling collimated displays may lead 
pilots to believe and act upon the resulting (and 
faulty) information about depth being provided 
by the accommodation/ convergence depth cues. 
This may be a particular problem during close 
encounter situations such as docking at gates. 
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Figure 3. Views of 13 ft. x 40 ft. truck located 25 ft. in front of pilot on taxiway. The two side views 
are through optical windows and the middle view is from a non-headslaved virtual window, with the 
sensor viewpoint located 2 ft. from, and directly in front of the virtual window. The top panel shows 
the view when the observer is in the center position (no eyepoint displacement). The middle panel is 
the view from the captains position (eyepoint displaced 1 ft. to the left). The bottom panel is the view 
from the center position, but with a 30% minification of the center virtual window. 


Display Minification 

This issue refers to the fact the display may be 
minified in order to fit more virtual space into 
some restricted display area (see Figure 3, bottom 
panel). When a display is minified apparent 
distance changes. Just like many side view mirrors 
on automobiles, the display should be labeled with 
“OBJECTS IN YOUR DISPLAY ARE CLOSER 
THAN THEY APPEAR”. But this is not the only 
effect, with concomitant distortions in apparent 
velocity and even in the apparent slant of surfaces 
(or in glideslope). 


Virtual/Optical Window Conflict 

This issue is, in many ways, the most complex and 
dangerous one of all since all of the above issues 
feed into it. Issue 7 refers to the fact that the 
physiological side effects of disorientation can 
arise when both a synthetic display and a real 
window are present on a flight deck, and no 
consistent visual interpretation is available. For 
example: 

1) Viewpoint Displacement: Even if 

displacement or a camera is not sufficient 
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to generate a significant visual-vestibular 
decoupling, the displacements may 
generate strong visual discrepancies 
between the window and the display 
during ground operations. For example, 
vertical displacements of the camera will 
be generate discrepancies in alignment of 
the imagery in the display and the real 
windows that are a function of the ratio of 
camera height to the height of the pilot 
above the ground. These discrepancies 
include shifts in apparent position (Figure 
3, middle panel) and in apparent speed of 
motion. 

2) Pilot Eyepoint Displacement: Vertical 
motions of the head will cause the scene 
in the window to transform, but unless 
head tracking is incorporated, the 
transformations will not be reflected in 
the display. Thus there is visual-vestibular 
coupling in one location and not in the 
other, which in turn may impact both 
compellingness and (dis)orientation. 

3) Stereo Vision Anomalies: Lack of 
binocular disparity in the display will 
produce a very different impression than 
looking through a true window. For 
surface operations where objects are truly 
close and within the range of stereo 
vision, this may interfere with the 
compellingness of the display as a virtual 
window. 

4) Display Displacement: Vertical 
displacements (which are the most likely) 
will strongly decouple the windows from 
the display. As with the above, this may 
interfere with the compellingness of the 
display, as well as lead to disorientation 
and difficulties in determining the spatial 
correspondence between the objects as 
they pass between the views of the display 
and the optical windows. 

5) Inappropriate 

Convergence/Accommodation Cues: The 
side window will have full convergence 


accommodation cues. How this will affect 
the pilot will probably depend on how 
peripheral vision is affected by these cues. 

6) Display Minification: Minification will 
naturally result in angular displacements, 
and angular size differences, between the 
similar sized and located objects in the 
display and in the optical windows. All of 
the potential problems noted for 
displacement of display surface from 
appropriate location will apply. In 
addition disorientation that may also 
result from the contrast between the 
distance/speed information distortions 
caused by minification and the natural 
view in the windows. 

Finally, perhaps the most dangerous aspect of the 
optical window issue is that we do not know how 
well a pilot can focus on the central display and 
ignore optical windows during an emergency 
situation where quick reactions are involved. The 
bottom two panels of Figure 3 show how side 
optical windows and a fixed perspective center 
display could generate clashing optical geometries. 
If the differences between the side optical and a 
display introduce some element of visual 
confusion, then this could result in longer 
response times while the pilot resolves the conflict. 

Dynamic Conformality 

Dynamic conformality is actually a variety of 
geometric conformality since it also is distorted 
only when the locations of the imaged objects 
within the virtual space do not overlap the optical 
locations of the real objects when directly viewed 
by the observer. However, dynamic conformality 
deserves to be treated separately as a result of its 
special character, i.e. it can be achieved only when 
the events in the physical space, and their 
depiction in the virtual space, occur 
simultaneously. The main disruptions in dynamic 
conformity are due to lags between when a scene 
should be updated and when it actually is updated. 
These lags, in turn, are due to update rate (frames 
per second being displayed) and to overall system 
lag (the time it takes to determine what should be 
displayed, and then to actually display it). The 
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update rate determines part of the overall system 
lag, but does not fully determine it. For example, a 
system may depend upon a camera that samples 
images 50 times a second, every 20 ms. In 
addition, however, it may take another 100 msec to 
process and display this image. This will generate 
a full 120 ms lag. 

While there are several factors determining how 
fast a virtual window must be updated, it must be 
fast enough that the displayed motion appears 
smooth. This is a function of update rate, the 
angular visual velocity being generated, and 
whether the image is discrete or time-sampled (i.e. 
motion-blurred). A displayed motion is most 
smooth when the angular distance between 
successive updates (angular velocity/update rate) is 
sufficiently small that smooth apparent motion is 
obtained. The required update rate will not need to 
be as high if the virtual window is displayed on a 
fixed panel, and not within an HMD. This is 
because the optical pattern within the virtual 
window is invariant over changes in the pilot’s 
gaze direction, but the location of the virtual 
window in retinal space does change. For a panel- 
mounted display the change in virtual window 
location in retinal space occurs automatically when 
head orientation changes, whereas it must be 
computed and updated when it is being drawn on 
an HMD. This can be a very significant factor 
since head rotation rates can generate angular 
motions in excess of 100 degrees/sec, much higher 
than the angular motions that result from simple 
translational head shifts. 

Another significant factor in determining required 
update rate is the likely angular visual velocities 
that must be displayed as a function of vehicular 
motion. These angular velocities are affected by 
three factors. The first factor is the angular rate of 
craft rotation during turns, which introduces 
angular optical velocities. The other two factors 
are functions of vehicle translation, and are 1) 
vehicle path speed divided by the range to the 
viewed object; and 2) the relative angle between 
the line of sight to the object and the velocity 
vector of the aircraft. Since surface movements do 
not often generate high rates of vehicle rotation, 
this factor can probably be safely ignored when 


determining required update rates. On the other 
hand, landing speeds on the order of 150 knots 
are expected for the high-speed transport aircraft 
of the future. Range to the viewed object is 
somewhat more difficult to assess since it will 
depend on the location of display viewpoint (e.g. 
camera location if a camera is generating the 
display, or pilot location if computer-generated 
imagery is being used to simulate the window that 
a pilot could be expected to see-through the 
virtual window), and the FOV. However, if we 
assume a fully conformal standard window that 
has a lookdown angle of 20 degrees, with the 
display viewpoint located 25 ft above the 
runway/taxiway, this will generate angular motions 
of 67 deg/sec at the bottom of the virtual window. 
Although a 60 Hz update rate mav provide an 
adequate display for this situation, locating a 
display viewpoint (e.g. camera or CGI viewport) 
below this will cause the angular motions to rise 
proportionately (e.g. the angular motions would 
be 134 deg/sec if the display viewpoint was at 12.5 
ft). And using a vertical FOV/viewpoint orientation 
with a lookdown angle of 30 degrees at 25 ft 
would cause the angular rate to rise to 144 deg/sec. 
Thus it is clear that the required update rate is 
strongly influenced by the placement and 
orientation of the display viewpoint. 

Dragging, the perception that a display is not 
keeping up with head motion, is due to system lags 
which delay the update of the display. When the 
display lag is coupled with independent (non- 
display), and non-lagged, information that the 
head has changed location. Therefore, apparent 
dragging could be expected to be most salient 
when the non-lagged information for head 
movement is salient, and/or the system lag is high. 
For a virtual window displayed in an HMD, only 
the vestibular system provides this independent 
information that the head has moved; while for a 
panel-mounted virtual window the change in the 
visible flight deck layout due to change in head 
location also provides this independent 
information. Therefore apparent dragging may be 
a more significant problem for panel-mounted 
virtual windows than for those displayed within an 
immersive HMD. Similarly, when synthetic scene- 
linked imagery is overlaid on a HUD, distortions 
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due to lags will be very obvious. In addition, it 
should be noted that the total system lags or 
transport delays often considered acceptable for 
flight simulation (often put at around 100-200 
msec) should not be extrapolated to form the 
bases for acceptable visual-visual or visual- 
vestibular lags, since inter-sensory/intra-sensory 
discrepancies should be much more noticeable. 

Finally, it may be possible to use predictive 
algorithms to compensate for some of the lags 
present in these systems. Such algorithms, which 
extrapolate over the expected lag period, are used 
in simulators to compensate for system lags. When 
using CGI imagery based upon a known database 
it is possible to use such predictive algorithms to 
remove lags in displays by using your present 
rates of change to estimate the expected future 
position that compensates for your lag. However, 
when the scene within the virtual window is based 
upon an image-based sensor, such as an optical or 
infrared camera, there is no independent database 
that can be used to describe what the scene will 
look like after you have moved by the expected 
amount and in the expected direction. Therefore, 
these algorithms can only be used if a suitable 
image warping technology is available. 

Conclusion 

For any standard application upon a flight deck, 
making a geometrically and dynamically correct 
virtual window will likely require a range of 
technologies (head-tracking, image-warping, 
displays separately viewable by the crew members, 
etc.). This is because a flight deck contains a crew, 
not just a single pilot. In addition, both crew 
members move their heads, especially during taxi 
operations, and any perspective display emulates 
the view from a true window only from a single 
eye position. Therefore, allowing dynamic head 
movements requires feeding this position back 
into the display generation system in a timely 
manner. This requires a system that has a 
sufficiently low lag, sufficiently high update rate, 
and a sufficiently high degree of accuracy. It is 
most likely that any system that is implemented in 
the foreseeable future will remain constrained by 
limitations in technology, and by other impinging 
flight deck design criteria (such constraints on 


available space for displays or the desire for 
optical windows in the aircraft). Preparation for 
implementing these designs requires, therefore, a 
systematic evaluation of the impacts of display 
conformality compromises. 
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